Skip to content

fix(nemotron_3.5_super): pin a vllm-router build that also fixes pref… - #3080

Open
sdevare-nv wants to merge 1 commit into
mainfrom
sdd/vllm-router-prefill-affinity
Open

fix(nemotron_3.5_super): pin a vllm-router build that also fixes pref…#3080
sdevare-nv wants to merge 1 commit into
mainfrom
sdd/vllm-router-prefill-affinity

Conversation

@sdevare-nv

Copy link
Copy Markdown
Contributor

What does this PR do?

Repins the vllm-router build used by the Super 3.5 eval container.

build_vllm_router_wheel.sh currently pins the head of
vllm-project/router#216, which
fixes the decode-node hot spot (issue #197). Shipping #216 on its own turns out to
be worse than not shipping it at all for prefill-heavy benchmarks like SWE-bench
Pro.

#216 makes the worker load counters honest, and that switches on a second latent
bug. cache_aware decides whether to use prefix affinity from the fleet-wide
load spread, so a single hot worker discards affinity for every request —
including requests whose own worker is idle. Under P/D disaggregation that gate is
open almost permanently, because prefill worker load counts queued requests as well
as running ones. Routing degenerates to shortest-queue, prompts that were already
cached get recomputed, prefill saturates, and decode starves behind it.

The pin now points at a branch carrying #216 plus a fix that applies the same
load check per request — against the worker the request actually wants, rather than
against the fleet maximum:
vllm-project/router#238.

Both pins are plain commit SHAs fetched from vllm-project/router — GitHub serves
any commit reachable from a ref, and a PR head is a ref even when the branch lives on
a contributor's fork. No fork remote is involved. Repin to a released commit once
these land upstream.

Also in the wheel build:

  • VLLM_ROUTER_SRC builds from a local router checkout instead of the pinned SHA,
    for iterating on a fork. Mounted read-only and copied into the build root, so the
    build never writes into the source tree.
  • cargo check --all-targets. Neither the wheel build nor cargo test --lib
    compiles the vllm-router binary target, so a break in src/main.rs shipped
    silently before.
  • The cache_aware policy tests run alongside the existing load-accounting ones.

Heads-up for reviewers

This pins an unmerged PR head, exactly as the existing #216 pin already does. Happy
to hold until #238 merges upstream if you would rather not carry an unmerged pin.

Testing

Validated on SWE-bench Pro (4 prefill / 6 decode) against two baselines on the same
checkpoint and vLLM config: the pre-#216 router, and #216 alone. With this pin the
prefill prefix-cache hit rate and end-to-end throughput return to pre-#216
behaviour while keeping #216's decode hot-spot fix.

…ill affinity

The wheel build pins the head of vllm-project/router#216, which fixes the
decode-node hot spot (issue #197). Shipping #216 alone turns out to be worse than
not shipping it for prefill-heavy benchmarks such as SWE-bench Pro.

#216 makes the worker load counters honest, and that switches on a second latent
bug: cache_aware decides whether to use prefix affinity from the fleet-wide load
spread, so one hot worker discards affinity for every request -- including requests
whose own worker is idle. Under P/D disaggregation that gate is open almost
permanently, because prefill worker load counts queued requests as well as running
ones. Routing degenerates to shortest-queue, already-cached prompts get recomputed,
prefill saturates and decode starves behind it.

Repin to the head of vllm-project/router#238, which carries #216 unchanged plus a
fix applying the same load check per request, against the worker the request
actually wants, rather than against the fleet maximum. Still fetched from
vllm-project/router by bare SHA, exactly as the #216 pin already was.

Signed-off-by: Sugam Devare <sdevare@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant